Control method, control apparatus, and program

ABSTRACT

There is provided a control method, including: outputting a displacement value depending on a displacement amount of an input device in a case where operation is input in the input device, the operation being for a control target displayed on a display apparatus, the position of the display apparatus being specified by a first coordinate system, the position of the input device being specified by a second coordinate system; calculating a displacement amount of the input device in the second coordinate system based on the displacement value; and displaying the control target, the control target being moved to a position corresponding to a displacement amount in the first coordinate system by using a transformation matrix, the transformation matrix transforming a displacement amount of the input device in the second coordinate system into a displacement amount in the first coordinate system.

TECHNICAL FIELD

The present disclosure relates to a control method including inputtingoperation for a control target, which is two-dimensionally orthree-dimensionally displayed on a display. The present disclosurefurther relates to a control apparatus and a program.

BACKGROUND ART

For example, mouse is widely used as input device for controlling GUIs(Graphical User Interfaces), which are two-dimensionally displayed on adisplay. Recently, in addition to 2D (two-dimensional) operation typeinput devices such as mouse, there are proposed many kinds of inputdevices, which may be operated in a 3D (three-dimensional) space.

Here, Patent Literature 1 discloses an input device configured to detectsextic movement, which is generated from movement of a mouse in a 3Dspace. The input device includes three acceleration sensors and threeangular velocity sensors. The three acceleration sensors detectacceleration in three axes, respectively. The three angular velocitysensors detect rotation around the three axes, respectively. The inputdevice transmits acceleration, velocity, relative position of the inputdevice, and posture of the input device to a computer. The computerprocesses such information. As a result, it is possible to control athree-dimensionally displayed control target.

CITATION LIST Patent Literature

[PTL 1]

Japanese Patent Application Laid-open No. H06-501119

SUMMARY Technical Problem

Meanwhile, a 3D space operation type input device includes accelerationsensors and angular velocity sensors. A user uses the input device tothereby control a control target. In this case, when the user startscontrol, it is necessary to conform axes specific to the respectivesensors to axes of the control target. This means the followingsituation. That is, in a case where a user holds the input device andstarts control or correction of errors, it is necessary for him to checkhis hand and to cause the input device to face a predetermined directionevery time.

Further, a display apparatus, which three-dimensionally displays animage, allows a user to recognize 3D information due to trick of theeye. Because of this, if a user frequently changes the direction of hisgaze to check his hand, he may feel tired. Further, if a user checks hishand and at the same time controls a control target displayed on adisplay, operation intuitiveness is decreased significantly. Because ofthis, it is necessary to naturally conform movement of a control targetto movement of an input device at hand. It is necessary to transformvalues (hereinafter, referred to as “displacement values”), which areoutput from sensors provided in an input device, to travel distance of acontrol target. In the case of transformation, calculation errors arelikely to accumulate. Here, if calculation errors accumulate, a gapbetween movement of the input device and movement of the control targetmay be generated. As a result, user-friendliness of the input device isdecreased significantly.

In view of the above-mentioned circumstances, it is desirable to providean input device capable of controlling a two-dimensionally orthree-dimensionally displayed control target even if a user does notcheck his hand and does not cause the input device to face apredetermined direction.

Solution to Problem

According to an embodiment of the present disclosure, there is provideda control method, including: outputting a displacement value dependingon a displacement amount of an input device in a case where operation isinput in the input device, the operation being for a control targetdisplayed on a display apparatus, the position of the display apparatusbeing specified by a first coordinate system, the position of the inputdevice being specified by a second coordinate system; calculating adisplacement amount of the input device in the second coordinate systembased on the displacement value; and displaying the control target, thecontrol target being moved to a position corresponding to a displacementamount in the first coordinate system by using a transformation matrix,the transformation matrix transforming a displacement amount of theinput device in the second coordinate system into a displacement amountin the first coordinate system.

According to the embodiment of the present disclosure, it is possible tocontrol a control target depending on movement of an input device.

Advantageous Effects of Invention

According to the embodiment of the present disclosure, the controlmethod includes displaying the control target, the control target beingmoved to a position corresponding to a displacement amount in the firstcoordinate system by using a transformation matrix, the transformationmatrix transforming a displacement amount of the input device in thesecond coordinate system into a displacement amount in the firstcoordinate system. Because of this, a user is capable of naturallymoving the control target in a direction that the user wishes to controlthe control target by using the input device. User-friendliness of theinput device, which has no directionality, is improved.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a configuration example of a controlsystem according to an embodiment of the present disclosure, the controlsystem including an input device, a control apparatus, and a displayapparatus.

FIG. 2 is an explanatory diagram showing a state where a user grips theinput device according to the embodiment of the present disclosure.

FIG. 3 is a partial cross-sectional view showing the input deviceaccording to the embodiment of the present disclosure.

FIG. 4 is a partial enlarged view of the cross-sectional view of FIG. 3.

FIG. 5 are external views each showing a shell part of the input deviceaccording to the embodiment of the present disclosure.

FIG. 6 is a block diagram showing the electric configuration of thecontrol system according to the embodiment of the present disclosure.

FIG. 7 is a functional configuration diagram showing the control systemaccording to the embodiment of the present disclosure.

FIG. 8 are flowcharts showing a behavior example of the control systemaccording to the embodiment of the present disclosure.

FIG. 9 is an explanatory diagram showing a control example of a controltarget, which is three-dimensionally displayed, in a case where a usermoves the input device according to the embodiment of the presentdisclosure in the depth direction of the display.

FIG. 10 is an explanatory diagram showing a control example of thecontrol target, which is three-dimensionally displayed, in a case wherea user rotates the input device according to the embodiment of thepresent disclosure.

FIG. 11 is an explanatory diagram showing a control example of thecontrol target, which is two-dimensionally displayed, in a case where auser rotates the input device according to the embodiment of the presentdisclosure.

FIG. 12 is an explanatory diagram showing definitions of a globalcoordinate system and a local coordinate system according to theembodiment of the present disclosure.

FIG. 13 is a line diagram showing relation between the local coordinatesystem and the global coordinate system according to the embodiment ofthe present disclosure.

FIG. 14 is an explanatory diagram showing an example of x, y, z axes ofacceleration sensors, angular velocity sensors, and magnetic sensorsmounted on the input device according to the embodiment of the presentdisclosure.

FIG. 15 is a flowchart showing an example of processing for conformingmovement of the control target to operation of the input device based ondisplacement values output from the various sensors, in a case where thecontrol apparatus according to the embodiment of the present disclosurereceives the displacement values from the input device.

FIG. 16 is an explanatory diagram showing an example of acceleration a′and angular velocity omega′ detected in the local coordinate systemaccording to the embodiment of the present disclosure.

FIG. 17 is an explanatory diagram showing an example of acceleration aand angular velocity omega transformed into the global coordinate systemaccording to the embodiment of the present disclosure.

FIG. 18 is an explanatory diagram showing an example of unit vectors i,j, k in the global coordinate system according to the embodiment of thepresent disclosure.

FIG. 19 is an explanatory diagram showing positional relation of a userand the display apparatus in the global coordinate system according tothe embodiment of the present disclosure.

FIG. 20 is an explanatory diagram showing an example of user's parallaxin addition to the positional relation of FIG. 19.

FIG. 21 is an explanatory diagram in a case where the control targetaccording to the embodiment of the present disclosure is displayed as ifthe control target bursts from the display apparatus.

FIG. 22 are block diagrams showing an internal configuration example ofa display apparatus according to another embodiment of the presentdisclosure.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described indetail with reference to the attached drawings. Note that, in thepresent description and drawings, the same reference symbols areattached to structural elements having substantially the same functionsand configurations, and repetition in descriptions thereof are avoided.A computer executes programs, whereby an input device realizes a controlmethod. Internal blocks (described below) execute the control method incooperation with each other.

1. Embodiment (example of transforming values in local coordinate systeminto global coordinate system)2. Modification examples

1. Embodiment Entire Configuration of System

First, with reference to FIG. 1, a control system according to anembodiment of the present disclosure will be described.

FIG. 1 shows a control system 100 including an input device 10 accordingto an embodiment and embodiments (described below) of the presentdisclosure. The control system 100 includes the input device 10, acontrol apparatus 50, and a display apparatus 60.

The input device 10 is a device having a spherical shape. A user inputsoperation in order to control the control target 60 b, for example. Thecontrol target 60 b is displayed on a display of a display unit 60 a.The control apparatus 50 controls the displayed control target 60 b inresponse to operation input in the input device 10.

The control apparatus 50 may be an apparatus dedicated for the inputdevice 10, or may be a PC or the like. In this manner, a user operatesthe input device 10 of the control system 100, to thereby remote controlthe control target 60 b displayed on the display unit 60 a.

Note that the display unit 60 a includes, for example, a liquid crystaldisplay, an EL (Electro-Luminescence) display, or the like. The displayunit 60 a may display a 2D image or may display a 3D image. The displayunit 60 a two-dimensionally or three-dimensionally displays the controltarget 60 b, which is controlled by the input device 10.

The control target 60 b such as a GUI is two-dimensionally displayed onthe display unit 60 a. Examples of such a GUI include, for example, apointer, an icon, a window, and the like. The control target 60 b suchas a human-shaped or animal-shaped character image isthree-dimensionally displayed. Note that those are merely examples. Thecontrol target 60 b may be any image as long as it is two-dimensionallyor three-dimensionally displayed.

Further, the display apparatus 60 may be a television apparatus capableof receiving television broadcasting and the like. Alternatively, thedisplay apparatus 60 displays the control target 60 bthree-dimensionally. In this case, the display apparatus 60 may be astereoscopic image display apparatus, which displays a stereoscopicimage such that a user is capable of visually recognizing thestereoscopic image with the naked eye. FIG. 1 shows a case where thecontrol apparatus 50 is independent from the display apparatus 60.Alternatively, an all-in-one apparatus including the control apparatus50 and the display apparatus 60 may be used. Further, the displayapparatus, which is configured to display the control target 60 b, maybe a projector or the like. In this case, a projector may control thecontrol target 60 b, which is projected on a screen or a wall surface.

The position of the display apparatus 60 is specified by a globalcoordinate system, i.e., a first coordinate system (described below).The position of the input device 10 is specified by a local coordinatesystem, i.e., a second coordinate system (described below). Further, auser inputs operation for controlling the control target 60 b in theinput device 10. Then, the control apparatus 50 controls the displayapparatus 60 such that the control target 60 b moves by a displacementamount of the input device 10 and is displayed.

(Structural Example of Input Device)

FIG. 2 is a diagram showing a state where a user grips the input device10. As shown in FIG. 2, the input device 10 has a spherical shape. Theinput device 10 is slightly larger or slightly smaller than a hardballof baseball. The diameter of the input device 10 is, for example, about50 mm to 100 mm Since the input device 10 has such a size, a user iscapable of handling the input device 10 easily when he grips the inputdevice 10. Note that the diameter of the input device 10 is not limitedto the above-mentioned range. As a matter of course, the diameter of theinput device 10 may be another value.

FIG. 3 is a partial cross-sectional view showing the input device 10.FIG. 4 is an enlarged view showing a part of the cross-sectional view ofFIG. 3. Further, each of FIGS. 5A and 5B is an external view showing ashell part 22 of the input device 10. FIG. 5A shows a state where theshell part 22 is seen from the oblique upward direction. FIG. 5B shows astate where the shell part 22 is seen from the A direction of FIG. 5A.

The input device 10 includes an input device main body 20. The inputdevice main body 20 includes a base part 21, a shell part 22, and agripper part 23. The base part 21 is provided at the center portion ofthe input device 10, and has a spherical shape. The shell part 22 isprovided so as to cover the whole surface of the base part 21, and has aspherical shell shape. The gripper part 23 is provided so as to coverthe whole surface of the shell part 22. The outline of the input devicemain body 20 is a sphere or an approximate sphere. Operation forcontrolling the control target 60 b, which is displayed on the displayapparatus 60, is input in the input device main body 20.

Further, the input device 10 includes tactile switches 12 (switch unit).The tactile switches 12 detect that the input device 10 is gripped withforce equal to or larger than a predetermined force, and gives a user aclick feeling. Further, the input device 10 includes pressure-sensitivesensors 13 (pressure sensors, which detect grip strength). Thepressure-sensitive sensors 13 detect grip strength when a user grips theinput device 10.

The base part 21 has a hollow inside. A circuit board is provided in thehollow portion in the base part 21. Electronic components including aCPU 11 and the like are mounted on the circuit board.

The shell part 22 includes eight plates 25 having the same shape (seeFIGS. 5A and 5B). The shape of each plate 25 is an approximate regulartriangle. Further, vertexes of corner portions of adjacent four plates25 out of the eight plates 25 meet at one point. In total, there are sixpoints, at which vertexes meet. The tactile switches 12 and thepressure-sensitive sensors 13 are arranged at positions corresponding tothe six points, respectively. That is, the input device 10 of thisembodiment includes six tactile switches 12 and six pressure-sensitivesensors 13. The tactile switches 12 and the pressure-sensitive sensors13 are provided between the surface of the base part 21 and the innersurface of the shell part 22 (plates 25) (see FIG. 3 and FIG. 4).

The pressure-sensitive sensors 13 are provided on the surface of thebase part 21. The tactile switches 12 are provided on thepressure-sensitive sensors 13, respectively. A first pressure diffusionplate 13 a intervenes between the pressure-sensitive sensor 13 and thetactile switch 12. A second pressure diffusion plate 13 b intervenesbetween the tactile switch 12 and the inner surface of the shell part 22(plates 25). Transmitting grip strength is generated when a user gripsthe gripper part 23. The first pressure diffusion plate 13 a and thesecond pressure diffusion plate 13 b are capable of transmitting thegrip strength to the pressure-sensitive sensor 13 uniformly. Thepressure-sensitive sensor 13 senses grip strength when a user grips theinput device main body 20.

The tactile switch 12 includes a switch main body 12 a and a movablepart 12 b. The movable part 12 b is capable of moving with respect tothe switch main body 12 a. Further, the tactile switch 12 includes anelectric switch mechanism (not shown) inside. The electric switchmechanism is turned on/off in response to movement of the movable part12 b. Further, the tactile switch 12 includes click feeling generatingmechanism (not shown). The click feeling generating mechanism employs anelastic body such as a plate spring. Such an elastic body generatesclick feeling in response to movement of the movable part 12 b.

Here, the relation between the pressure-sensitive sensors 13 and forceapplied to the plate 25 will be described. Force applied to one plate 25and a force-applied position are calculated based on pressure valuesdetected by the pressure-sensitive sensors 13. In this case, at leastthree pressure-sensitive sensors 13 are required for one plate 25.

In this embodiment, three pressure-sensitive sensors 13 are provided forone plate 25. The three pressure-sensitive sensors 13 detect forceapplied to the plate 25 (this plate 25 and other plates 25 sharepressure-sensitive sensors 13). With this structure, calculationemploying vector calculation or the like is executed based on pressurevalues output from the pressure-sensitive sensors 13. As a result, it ispossible to calculate force applied to the plate 25 and a force-appliedposition accurately.

Further, three pressure-sensitive sensors 13 are used for each of theeight plates 25. In this case, essentially, two dozens ofpressure-sensitive sensors 13 are required (8×3=24). However, accordingto this embodiment, the pressure-sensitive sensor 13 is arranged at aposition at which vertexes of corner portions of adjacent four plates 25meet. The adjacent four plates 25 share one pressure-sensitive sensor13.

Because of this structure, the six pressure-sensitive sensors 13 areenough in total. It is possible to reduce cost of the input device 10.As described above, it is possible to accurately calculate force appliedto the plate 25 and a force-applied position with the minimum number ofpressure-sensitive sensors 13.

However, the pressure-sensitive sensors 13 may not always have theabove-mentioned structure. For example, one pressure-sensitive sensor13, two pressure-sensitive sensors 13, or four or morepressure-sensitive sensors 13 may be provided for one plate 25. Further,the plates 25 may not share the pressure-sensitive sensor 13. That is,the pressure-sensitive sensors 13 may be provided for the plates 25,respectively.

Typically, the pressure-sensitive sensors 13 may employ any mode as longas the pressure-sensitive sensors 13 are capable of detecting forceapplied to the plates 25 (shell part 22) when a user grips the inputdevice 10. Further, the number of the plates 25 (division number ofshell part 22) may not be limited to eight. For example, the number ofthe plates 25 may be two, four, or the like.

Each of the base part 21 and the shell part 22 is made of materials suchas, for example, metal or resin. Meanwhile, the gripper part 23 is madeof a material softer than the base part 21 and the shell part 22. As amaterial of the gripper part 23, for example, sponge or the like, i.e.,foamed synthetic resin such as polyurethane, may be used. In a casewhere a material such as sponge is used as a material of the gripperpart 23, tactile impression may be improved. Further, a user is capableof slightly adjusting grip strength when he grips the input device 10.

Next, with reference to FIG. 6 and FIG. 7, the electric configuration ofthe control system 100 will be described.

FIG. 6 is a block diagram showing the electric configuration of thecontrol system 100.

FIG. 7 is a functional configuration diagram showing the control system100.

First, with reference to FIG. 6, the electric configuration of the inputdevice 10 will be described.

The input device 10 includes a notification mechanism 9, the CPU(Central Processing Unit) 11, the tactile switches 12, thepressure-sensitive sensors 13, acceleration sensors 14, angular velocitysensors 15, and magnetic sensors 16. Further, the input device 10includes a sending/receiving circuit 17, a battery 18, a ROM (Read OnlyMemory) 19 a, and a RAM (Random Access Memory) 19 b. Note that thenotification mechanism 9 may employ a light emitting mechanism such asan LED (Light Emitting Diode), a sound producing mechanism such as aspeaker, or a vibration generating mechanism. In this case, thenotification mechanism 9 may carry out notification by means of at leastone of sound producing, light emitting, and vibration generating.

The notification mechanism 9, the CPU 11, the acceleration sensors 14,the angular velocity sensors 15, the magnetic sensors 16, thesending/receiving circuit 17, the ROM 19 a, and the RAM 19 b are mountedon the circuit board (not shown). The circuit board, on which theelectric components are mounted, and the battery 18 are provided in thehollow portion in the base part 21. In this manner, the respectivesensors are accommodated in the input device main body 20.

Each of the acceleration sensors 14 and the angular velocity sensors 15is a sensor configured to detect movement of the input device 10 in the3D space. The acceleration sensors 14 detect acceleration in three-axisdirections orthogonal to each other. The acceleration sensors 14 outputacceleration values (example of displacement value) to the CPU 11. Theacceleration values depend on the detected acceleration. The angularvelocity sensors 15 detect angular velocity around the three axesorthogonal to each other. The angular velocity sensors 15 output angularvelocity values (example of displacement value) to the CPU 11.

The angular velocity values depend on the detected angular velocity.Further, the magnetic sensors 16 detect geomagnetic directions (forexample, magnetic north) of the input device 10 in the 3D space. Themagnetic sensors 16 output magnetic values (example of displacementvalue) to the CPU 11.

The tactile switches 12, the pressure-sensitive sensors 13, a touchsensor, and the like function as a pressure-sensitive unit. When theswitch mechanism is turned on, the tactile switch 12 outputs a signal tothe CPU 11. The pressure-sensitive sensor 13 is an example of a pressuresensor. Such a pressure sensor outputs a pressure value to the CPU 11.The pressure value depends on grip strength when a user grips the inputdevice 10.

The CPU 11 executes various kinds of calculation based on angularvelocity values output from the acceleration sensors 14, accelerationvalues output from the angular velocity sensors 15, magnetic valuesoutput from the magnetic sensors 16, and pressure values output from thepressure-sensitive sensors 13, in order to control the control target 60b. Further, operation for controlling the control target 60 b, which isdisplayed on the display apparatus 60, is input in the input device 10.

Then, the CPU 11 outputs displacement values depending on displacementamounts of the input device 10. The displacement amounts include, forexample, acceleration values, angular velocity values, magnetic values,and pressure values.

Further, the displacement amount is obtained based on displacementamounts of one axis, two axes orthogonal to each other, or three axesorthogonal to each other.

The CPU 11 sends the obtained acceleration values, angular velocityvalues, magnetic values, and pressure values from the sending/receivingcircuit 17 to the control apparatus 50. Note that the CPU 11 maycalculate grip strength when a user grips the input device 10, aforce-applied position, and the like based on the obtained accelerationvalues, angular velocity values, magnetic values, and pressure values.Specifically, the grip strength, the force-applied position, and thelike include a travel distance of the input device 10 in the space, arotational amount, an angle with respect to the magnetic north, and thelike. Note that the CPU 11 executes various kinds of calculation in astate where the switch mechanisms of the tactile switches 12 inputsignals in the CPU 11.

The sending/receiving circuit 17 includes an antenna (not shown) and thelike.

Controlled by the CPU 11, the sending/receiving circuit 17 sends variouskinds of information to the control apparatus 50. Note that thesending/receiving circuit 17 is also capable of receiving informationsent from the control apparatus 50.

A rechargeable secondary battery, for example, is used as the battery18. A charging unit (not shown) is accommodated in the input device mainbody 20.

The notification mechanism 9 notifies a user that the control target 60b is displaced. Here, the notification mechanism 9 may notify a user ofa displacement amount of travel of the control target 60 b with respectto each axis, independently.

Next, with reference to FIG. 7, the functional configuration of theinput device 10 will be described. The input device 10 includes adetection unit 43, a first processor unit 41, a storage unit 46, a firsttransmitting/receiving unit 47, and a power source unit 48. Thedetection unit 43 includes a first operation detection unit 44, and asecond operation detection unit 45. The first processor unit 41 includesa first response unit 42, a first calculation unit 40, and a secondresponse unit 49.

The detection unit 43 detects that the input device main body 20 isgripped by a user with force equal to or larger than a predeterminedforce. For example, the detection unit 43 detects grip strength when theinput device main body 20 is gripped. The detection unit 43 outputs thedetected grip strength value depending on the grip strength. The firstoperation detection unit 44 detects first operation, which is input inthe input device main body 20. The first operation is not limited togriping the input device 10. Examples of the first operation includegriping the input device 10, raising the input device 10 from a table,tapping the input device 10, and the like. The first operation detectionunit 44 may detect that, for example, the input device main body 20 isgripped with predetermined pressure (for example, first threshold) ormore. Further, the first operation detection unit 44 may output adetected grip strength value based on the detection result. The gripstrength value depends on grip strength.

After the first operation detection unit 44 detects the first operation,the second operation detection unit 45 detects second operation input inthe input device main body 20. The second operation detection unit 45detects second operation after the first operation detection unit 44detects the first operation. That is, the first operation and the secondoperation are not detected simultaneously.

The second operation is not limited to griping the input device 10.Examples of the second operation include grabbing and operating theinput device 10 (grabbing and shaking input device 10) and the like.Detected values, which are detected by the first operation detectionunit 44 and the second operation detection unit 45, may include valuessensed by at least one of the pressure-sensitive sensors 13, theacceleration sensors 14, the angular velocity sensors 15, and themagnetic sensors 16. The second operation detection unit 45 detectsthat, for example, the input device main body 20 is gripped withpressure equal to or larger than a predetermined pressure (secondthreshold). The second threshold, which has a value larger than thefirst threshold, is preset.

The first response unit 42 returns a first response from the inputdevice 10 based on the detected first operation. Examples of the firstresponse include producing click sound, returning bounce to the hand ofa user, who grips the input device 10, by means of a spring provided inthe input device 10, and the like. Note that the first response unit 42does not necessarily return a first response to a user based on detectedfirst operation. For example, the first response unit 42 may return afirst response to a user when the detection unit 43 detects that theinput device 10 is gripped, and when the detection unit 43 detects thatthe input device 10 is not gripped. The first response unit 42 mayreturn a first response to a user without being controlled by the firstprocessor unit 41, at least when the detection unit 43 detects that theinput device main body 20 is gripped. Further, basically, the firstresponse is immediately returned because the first response is returnedwithout using the CPU 11. Alternatively, the first response may bereturned by using the CPU 11.

The first calculation unit 40 may execute calculation (first processing)for controlling the control target 60 b based on a displacement valuewith respect to movement of the input device main body 20. The movementof the input device main body 20 depends on the first operation. Thefirst calculation unit 40 may execute calculation (second processing)for controlling the control target 60 b based on detected movement ofthe input device main body 20. The movement of the input device mainbody 20 depends on the second operation.

Note that the first operation detection unit 44 is capable of causingthe first response unit 42 to return a response without using the firstcalculation unit 40. Because of this, the first response unit 42 iscapable of returning a response, i.e., click sound, to a user withoutsending a first response to the first calculation unit 40.

As a result, it is possible to return a first response, i.e., clicksound, by the first response unit 42 faster than processing by the firstcalculation unit 40 based on a calculation result. However, theresponding process is not limited to this. Alternatively, a firstresponse (click sound, for example) may be sent to the first calculationunit 40. The first response may be used as a trigger to startcalculation by the first calculation unit 40.

Further, the first processing and the second processing are differentkinds of control to the control target 60 b, and do not involve the samekind of control. An example of the different kinds of control is asfollows. That is, as the first processing, click sound is generated in acase where grip strength is equal to or more than predetermined firstthreshold pressure when a user grips the input device 10. As the secondprocessing, calculation for controlling the control target 60 b isexecuted based on displacement values output from the various sensors ina case where a user grips the input device 10 more tightly and wheregrip strength is equal to or more than predetermined second thresholdpressure. Another example of the different kinds of control is asfollows. That is, as the first processing, in a case where a user gripsthe input device 10 for two seconds, an “enable” (operation input OK)status is established, and a cursor move mode is started. As the secondprocessing, in a case where a user grips the input device 10 again forfour seconds, the control target 60 b is selected, and a control mode isstarted. An example of the same kind of control is as follows. That is,the pressure-sensitive sensor 13 detects three levels of values. Thefirst processing is executed when a user grips the input device 10lightly. The second processing is executed when a user grips the inputdevice 10 tightly. The first processing and the second processing of theembodiments do not involve such a same kind of control.

Note that the first response unit 42 does not send the first response tothe control apparatus 50, which controls the control target 60 b. Thefirst response is for a user of the input device. That is, the controlapparatus 50 does not receive the first response from the first responseunit 42. The first calculation unit 40 sends a calculation result forcontrolling the control target 60 b to the control apparatus 50.

That is, the control apparatus 50 receives the calculation result fromthe first calculation unit 40. As described above, the first responseprocessing by the first response unit 42 involves a route including nosending/receiving process. As a result, the first response processing isfaster than processing based on a calculation result by the firstcalculation unit 40, which includes sending/receiving process, and isthus effective.

The second response unit 49 returns a second response from the inputdevice 10 based on a detected second operation. An example of the secondresponse is to give feedback to a user. Examples of the feedback are asfollows. That is, an LED of the notification mechanism 9 of the inputdevice 10 blinks, the notification mechanism 9 outputs sound, thenotification mechanism 9 causes a user to feel a sense of force, and thenotification mechanism 9 vibrates. Basically, the second response isimmediately returned because the second response is returned withoutusing the CPU 11. Alternatively, the second response may be returned byusing the CPU 11.

Further, as necessary, both of or one of the first processor unit 41 anda second processor unit 61 of the control apparatus side execute desiredprocess based on detection results, and outputs the results to a displaycontroller unit 66 via the second processor unit 61.

The storage unit 46 may be realized by the ROM 19 a or the RAM 19 bincluding, for example, a semiconductor memory, a magnetic disk, anoptical disk, or the like.

The first transmitting/receiving unit 47 sends or receives predeterminedinformation between the input device 10 and the control apparatus 50.The first transmitting/receiving unit 47 is wired or wireless-connectedto a second transmitting/receiving unit 62. The power source unit 48includes, for example, a rechargeable battery as the battery 18, andsupplies electric power to the respective units.

(Configuration of Control Apparatus)

Next, with reference to FIG. 6, the electric configuration of thecontrol apparatus 50 will be described.

The control apparatus 50 includes a CPU 51, a sending/receiving circuit52, a ROM 53 a, a RAM 53 b, and an instruction mechanism 54.

The ROM 53 a is a nonvolatile memory, and stores various programsnecessary for processing executed by the CPU 51. The RAM 53 b is avolatile memory, and is used as a work area for the CPU 51.

The sending/receiving circuit 52 includes an antenna and the like, andreceives various kinds of information sent from the input device 10.Further, the sending/receiving circuit 52 is also capable of sending asignal to the input device 10.

The instruction mechanism 54 is, for example, a keyboard. A user inputssetting such as initial setting and special setting by using theinstruction mechanism 54.

The instruction mechanism 54 receives various instructions from a user,and outputs input signals to the CPU 51.

The CPU 51 executes functions of a second processor unit (describedbelow). The second processor unit executes processing of a secondcalculation unit and processing of a display controller unit. The CPU 51controls a control target, which is displayed on the display apparatus60, based on various kinds of information received by thesending/receiving circuit 17.

Next, with reference to FIG. 7, the functional configuration of thecontrol apparatus 50 will be described.

The control apparatus 50 includes the second processor unit 61, astorage unit 63, the second transmitting/receiving unit 62, and aninstruction unit 64. The second processor unit 61 includes a secondcalculation unit 65, and the display controller unit 66.

The second transmitting/receiving unit 62 sends/receives predeterminedinformation to/from the first transmitting/receiving unit 47. Thestorage unit 63 may be realized by the ROM 53 a or the RAM 53 bincluding, for example, a semiconductor memory, a magnetic disk, anoptical disk, or the like.

For example, in a case where a user inputs operation by using theinstruction mechanism 54 such as a keyboard, the instruction unit 64executes setting such as initial setting and special setting.Specifically, the instruction unit 64 receives various instructions froma user, outputs the input signals to the second processor unit 61, andinstructs the instruction unit 64 to execute initial setting and thelike.

The second calculation unit 65 executes desired processing based on adetection result and a calculation result from the first processor unit41, and outputs the results to the display controller unit 66. Further,the second calculation unit 65 calculates a displacement amount of theinput device 10 in a local coordinate system (described below) based ona displacement value received from the input device 10. Note that thefirst calculation unit 40 of the input device 10 may calculate adisplacement amount of the input device 10, and may send the calculateddisplacement amount to the control apparatus 50.

The display controller unit 66 controls how to display the controltarget 60 b based on obtained information. Here, the display controllerunit 66 displays the control target 60 b as follows. That is, thedisplay controller unit 66 moves the control target 60 b to a positioncorresponding to a displacement amount in a global coordinate system byusing a transformation matrix. The transformation matrix is used totransform a displacement amount of the input device 10 in a localcoordinate system (described below) to a displacement amount in theglobal coordinate system. The transformation matrix is used to transformacceleration values and angular velocity values in a local coordinatesystem into acceleration and angular velocity of the control target 60 bin the global coordinate system. Note that the control apparatus 50 mayinclude an expression unit in place of the display controller unit 66 orin addition to the display controller unit 66. The expression unit iscapable of expressing an action (for example, sound, vibration, etc.)other than display of the control target 60 b. The expression unitcontrols expression of the control target 60 b in response to desiredinput operation based on obtained information.

(Behavior of Input Device)

Next, behavior of the control system 100 of this embodiment will bedescribed. FIGS. 8A and 8B are flowcharts showing a behavior example ofthe control system 100 of this embodiment. FIG. 8A shows processingexecuted by the input device 10. FIG. 8B shows processing executed bythe control apparatus 50.

The input device 10 sends all the values obtained from various sensorsto the control apparatus 50. Further, the control apparatus 50 executesintegral calculation and coordinate transformation based on the valuesreceived from the input device 10.

First, a user raises the input device 10, and moves the input device 10to a position at which he may operate the input device 10 easily. Notethat, in this case, the control target 60 b, which is displayed on thedisplay unit 60 a, does not move (see Step S1, No). Next, the user gripsthe gripper part 23 of the input device main body 20 with force equal toor larger than predetermined force in order to start to operate theinput device 10. Then, the shell part 22 (plates 25) of the input devicemain body 20 and the movable parts 12 b of the tactile switches 12 movein directions that they come close to the center of the input device 10.When the movable parts 12 b of the tactile switches 12 move indirections that they come close to the center of the input device 10,the click feeling generating mechanisms generate click feeling.

The input device 10 returns a response (example of first response),i.e., click feeling, whereby the input device 10 is capable ofappropriately returning a response to intention of a user to start tocontrol the control target 60 b. Then, a user is capable of easilyrecognizing, with the click feeling, that control of the control target60 b is started. Further, the click feeling generating mechanisms arecapable of returning a response, i.e., click feeling, to a userimmediately because the click feeling generating mechanisms return aresponse, i.e., click feeling, without using the CPU.

When the movable parts 12 b of the tactile switches 12 move indirections that they come close to the center of the input device 10,click feeling is generated. At the same time, the switch mechanisms ofthe tactile switches 12 are turned on, and the switch mechanisms inputsignals to the CPU 11 (Step S1, Yes).

When the tactile switches 12 input the signals, the CPU 11 obtainsacceleration values from the acceleration sensors 14, angular velocityvalues from the angular velocity sensors 15, and magnetic values fromthe magnetic sensors 16. The CPU 11 obtains pressure values from thepressure-sensitive sensors 13 (Step S2).

Next, the CPU 11 averages the acceleration values, the angular velocityvalues, and the magnetic values (Step S3). Further, the CPU 11 executescalculation based on the pressure values by means of vector calculationand the like. As a result, the CPU 11 calculates grip strength (forceapplied to plates 25) when a user grips the input device 10, and aforce-applied position.

Next, the CPU 11 sends the respective pieces of information (averageacceleration value, average angular velocity value, average magneticvalue, grip strength when user grips input device 10, and force-appliedposition) to the control apparatus 50 via the sending/receiving circuit17 (Step S4).

The CPU 51 of the control apparatus 50 determines if the respectivepieces of information are received from the input device 10 (Step S11).If the respective pieces of information are received from the inputdevice 10, the CPU 51 of the control apparatus 50 integrates theacceleration values and the angular velocity values out of therespective pieces of received information. As a result, the CPU 51obtains the travel distance and the rotational amount of the inputdevice 10. The CPU 51 controls the control target 60 b (Step S12). Notethat, in Step S12, the CPU 51 of the control apparatus 50 may furtherexecute calculation of the respective pieces of received information,and may execute process to improve accuracy of controlling the controltarget 60 b.

For example, the control target 60 b is a three-dimensionally displayedcharacter image. In this case, in Step S12, the CPU 51 causes thecharacter image to travel three-dimensionally and rotates the characterimage three-dimensionally based on information such as the traveldistance and the rotational amount of the input device 10. Further, theCPU 51 causes the character image to execute particular movement (forexample, jump, crouch, laugh, anger, etc.) based on information on gripstrength and information of a force-applied position. Note that how tocontrol the control target 60 b based on information on travel distance,rotational amount, grip strength, and a force-applied position, is notspecifically limited.

A user moves the input device 10, rotates the input device 10, grips theinput device 10 more tightly, or presses a particular position of theinput device 10 hard, while griping the input device 10 with force equalto or larger than predetermined force. As the result of process shown inFIGS. 8A and 8B, it is possible to move the control target 60 barbitrarily.

Meanwhile, a user wishes to (temporarily) stop controlling the controltarget 60 b. In this case, the user weakens the grip strength when hegrips the input device 10. A user weakens the grip strength when hegrips the input device 10, and the grip strength falls below thepredetermined force. Then, the movable parts 12 b of the tactileswitches 12 and the shell part 22 (plates 25) of the input device mainbody 20 move in directions that they depart from the center of the inputdevice 10.

When the movable parts 12 b of the tactile switches 12 move indirections that they depart from the center of the input device 10, theclick feeling generating mechanisms generate click feeling.

The input device 10 returns the response, i.e., click feeling, wherebythe input device 10 is capable of appropriately returning a response tointention of a user to stop controlling the control target 60 b. Then, auser is capable of easily recognizing, with the click feeling, thatcontrol of the control target 60 b is stopped. When the movable parts 12b of the tactile switches 12 move in directions that they depart fromthe center of the input device 10, click feeling is generated. Inaddition, the switch mechanisms of the tactile switches 12 stopoutputting signals. As a result, the tactile switches 12 stop inputtingsignals in the CPU 11 (Step S1, No), and the control target 60 b stopsmoving.

In this manner, a user grips the input device 10 with force equal to orlarger than the predetermined force, and weakens the grip strength whenhe grips the input device 10. As a result, the user is capable ofarbitrarily selecting one of reflecting operation (operation in space,operation based on grip strength) of the input device 10 in control ofthe control target 60 b and not reflecting operation of the input device10 in control of the control target 60 b.

Further, the click feeling generating mechanisms of the tactile switches12 of the input device 10 are capable of appropriately returning aresponse to intention of a user to start to control the control target60 b. Then, a user is capable of easily recognizing, with the clickfeeling, that control of the control target 60 b is started. Further,the click feeling generating mechanisms are capable of returning aresponse, i.e., click feeling, to a user immediately because the clickfeeling generating mechanisms return a response, i.e., click feeling,without using the CPU.

Further, the input device 10 is capable of rapidly returning a response,i.e., click feeling, whereby the input device 10 is capable of returninga response to intention of a user to stop controlling the control target60 b. Then, a user is capable of easily recognizing, with the clickfeeling, that control of the control target 60 b is stopped.

(Operational Example of Input Device)

Next, a control example of the control target 60 b by using the inputdevice 10 will be described.

FIG. 9 is an explanatory diagram showing a control example of thecontrol target 60 b, which is three-dimensionally displayed, in a casewhere a user moves the input device 10 in the depth direction of thedisplay.

Let's assume that a user moves his hand, which holds the input device10, in a direction (X direction) toward the display apparatus 60. Then,the control target 60 b is displayed such that the control target 60 bmoves in the depth direction (X direction) of the display of the displayapparatus 60, similar to the movement of the input device 10. Thismovement is expressed by means of a 3D image due to trick of the eye.The motion axes of the control target 60 b include not only X directionof FIG. 9 but also YZ directions. The motion axes of the control target60 b further include oblique directions, which are obtained bysynthesizing the three axes. Because of this, the control target 60 bmoves front/back/left/right/up/down together withfront/back/left/right/up/down movement of the input device 10irrespective of griping mode of the input device 10.

FIG. 10 is an explanatory diagram showing a control example of thecontrol target 60 b, which is three-dimensionally displayed, in a casewhere a user rotates the input device 10.

Similar to FIG. 9, if a user rotates the input device 10 around Y axis,the control target 60 b also rotates around Y axis. In this case also,movement of the control target 60 b does not depend on holding directionof the input device 10. If a user rotates the input device 10 around XYZaxes, the control target 60 b is displayed such that the control target60 b rotates around XYZ axes, irrespective of holding mode of the inputdevice 10 by a user. Further, the rotational axes include not only Ydirection of FIG. 10. The control target 60 b is capable of rotatingaround XZ directions in the same way. The rotational axes furtherinclude oblique directions, which are obtained by synthesizing the threeaxes.

FIG. 11 is an explanatory diagram showing a control example of thecontrol target 60 b, which is two-dimensionally displayed, in a casewhere a user rotates the input device 10.

The control target 60 b shown in FIG. 9 or FIG. 10 is a 3D image due totrick of the eye. However, as shown in FIG. 11, it is possible tocontrol the 2D-image control target 60 b having depth information in thesame way. In this case, similar to the control shown in FIG. 9 and FIG.10, a user moves/rotates the input device 10, which is gripped with hishand, in/around XYZ axes. Such travel/rotation control is reflected inmovement of the control target 60 b, which is two-dimensionallydisplayed.

Next, with reference to FIG. 12 and FIG. 13, definitions of coordinatesystems and axes will be described. The coordinate systems and the axesare used when mathematical formulae (described below) are described.

FIG. 12 is an explanatory diagram showing definitions of the globalcoordinate system and the local coordinate system.

Hereinafter, the coordinate system having a plane in parallel with afloor surface is expressed by the global coordinate system X, Y, Z(capital letters). The coordinate system specific to the input device 10is expressed by the local coordinate system x, y, z (italic smallletters). Further, the global coordinate system is used for acoordinate, which specifies the position of the control target 60 bdisplayed on the display of the display apparatus 60. For example, thevertical direction of the display is referred to as Z axis, the lateraldirection of the display is referred to as X axis, and the depthdirection of the display is referred to as Y axis. Further, the localcoordinate system is used for a coordinate, which specifies the inputdevice 10. The front direction of the input device 10 is referred to asy axis, the lateral direction of the input device 10 is referred to as xaxis, and the thickness direction of the input device 10 is referred toas z axis.

FIG. 13 is a line diagram showing relation between the local coordinatesystem and the global coordinate system.

Here, Y axis of the global coordinate system faces the magnetic northorientation N. It means that the display of the display apparatus 60faces north. A vector V is a vector showing the direction of the inputdevice 10, and is in parallel with y axis of the local coordinatesystem. A vector V₀ is obtained by projecting the vector V on the XYplane (horizontal plane (floor surface)), and is a vector on the XYplane. An orientation angle alpha corresponds to an angle formed by Yaxis and the vector V₀. A pitch angle phi corresponds to an angle formedby the vector V and the vector V₀, and is a rotation angle around x axisof the local coordinate. A roll angle theta is an angle around y axis onthe xy plane of the local coordinate, and corresponds to an intersectionangle of the yZ plane and the yz plane. A vector H is a vectorindicating the geomagnetic direction, and faces the magnetic pole N.

A geomagnetic depression/elevation angle theta_(H) is an angle formed bythe XY horizontal plane and the geomagnetic direction, and correspondsto an angle formed by Y axis and the vector H.

FIG. 14 is an explanatory diagram showing an example of x, y, z axes ofthe acceleration sensors 14, the angular velocity sensors 15, and themagnetic sensors 16 mounted on the input device 10.

FIG. 14 shows that the acceleration sensor 14, the angular velocitysensor 15, and the magnetic sensor 16 are arranged on each of xyz axesof the local coordinate system. That is, the three acceleration sensors14, the three angular velocity sensors 15, and the three magneticsensors 16 are arranged in the local coordinate system. Note that atleast one acceleration sensor 14, at least one angular velocity sensor15, and at least one magnetic sensor 16 may be provided.

FIG. 15 is a flowchart showing an example of processing for conformingmovement of the control target 60 b to operation of the input device 10based on displacement values output from the various sensors. Thecontrol apparatus 50 receives the displacement values from the inputdevice 10.

This embodiment realizes the input device 10 capable of conformingmovement of the control target 60 b to behavior of a user irrespectiveof directionality of the input device 10, when the two-dimensionally orthree-dimensionally displayed control target 60 b is controlled. Here,the CPU 51 of the control apparatus 50 transforms displacement values ofthe local coordinate system, which are output from the various sensors,into the global coordinate system. In this case, calculation is executedas shown in the flowchart of FIG. 15. As a result, calculation error isminimized.

The display apparatus 60 complements displacement values received fromthe acceleration sensors 14, the angular velocity sensors 15, and themagnetic sensors 16 arranged on the respective axes of the localcoordinate, to thereby execute calculation. Hereinafter, how to processdisplacement values obtained from the various sensors will be describedspecifically.

First, the CPU 51 receives acceleration values, angular velocity values,and magnetic values in xyz axes direction from the input device 10. TheCPU 51 determines if the input device 10 starts moving such as travel orrotation (Step S1). Then, the CPU 51 determines whether to proceed tothe subsequent calculation processing.

The CPU 51 determines if the input device 10 starts moving or not asfollows. Displacement values, which are output from the accelerationsensors 14 arranged on the local coordinate x, y, z axes, are referredto as X_(accl), Y_(accl), and Z_(accl), respectively. In this case, thefollowing mathematical formula (1) expresses acceleration.

Here, whether the input device 10 moves or not is determined based onwhether the acceleration is equal to the constant 1 (G) or not. That is,if acceleration obtained based on the mathematical formula (1) is equalto 1 (G), it is determined that the input device 10 moves.

If the mathematical formula (1) is not satisfied, it is understood thatmovement acceleration is applied to the input device 10 in addition togravity acceleration. As a result, the CPU 51 determines that the inputdevice 10 starts moving. Note that predetermined constants A₁, A₂ may bedefined, and the CPU 51 may determine if operation of the input device10 is started or not based on the following mathematical formula (2).

Here, for example, the constant A₁ is 0.9 (G), and the constant A₂ is1.1 (G). In this case, the CPU 51 does not determine that the inputdevice 10 starts moving even if the input device 10 slightly travels.Further, the CPU 51 determines that the input device 10 starts moving ifthe mathematical formula (2) is not satisfied.

Note that the CPU 51 may determine if the input device 10 starts movingby using displacement values other than the above-mentioned mathematicalformulae (1) and (2).

For example, displacement values X_(gyro), Y_(gyro), and Z_(gyro) outputfrom the angular velocity sensors 15 arranged on the local coordinate x,y, z axes are used. In this case, each of displacement values X_(gyro),Y_(gyro), and Z_(gyro), is 0 when the input device does not move. Inthis case, the CPU 51 substitutes the displacement values X_(gyro),Y_(gyro), and Z_(gyro), which are output from the angular velocitysensors 15 arranged on the local coordinate x, y, z axes, in X_(accl),Y_(accl), and Z_(accl) of the mathematical formulae (1) and (2),respectively. As a result, the CPU 51 is capable of determining if theinput device 10 moves or not. Further, the CPU 51 may substitutedisplacement values output from the magnetic sensors 16 in themathematical formulae (1) and (2), to thereby determine if the inputdevice 10 starts moving.

Further, the input device 10 may instruct the display apparatus 60 tostart moving not by using displacement values output from the varioussensors but by using operations input in the tactile switches 12 and thepressure-sensitive sensors 13. For example, let's assume that thetactile switches 12 are used. In this case, a user, who holds the inputdevice 10, presses the tactile switches 12 when he wishes to startcontrol. Then, the input device 10 may determine that thetravel/rotation status is started, and may start calculation. Asdescribed above, a method of determining if movement is started or notmay be realized by combining the various sensors and mechanisms.

Next, the CPU 51 determines inclination of the input device 10 held by auser in the global coordinate system based on gravity accelerationobtained from displacement values output from the acceleration sensors14 (Step S2). Here, with reference to the respective coordinate systemsof FIG. 13, an example of a method of calculating posture (localcoordinate system) of the input device 10 with respect to the floorsurface (global coordinate system) will be described.

First, a method of calculating the roll angle theta will be described.The roll angle theta is an angle around y axis on the local coordinatexy plane, and is an intersection angle of the yZ plane and the yz plane.The CPU 51 calculates the roll angle theta based on acceleration valuesin the local coordinate system received from the input device 10. Theinput device 10 faces the display apparatus 60, and rotates by the rollangle theta. Here, A is indicative of the displacement value output fromthe acceleration sensor 14 arranged on x axis with respect to gravityacceleration 1 G. X_(accl) is indicative of the displacement valuemeasured by the acceleration sensor 14 arranged on x axis. Z_(accl) isindicative of the displacement value measured by the acceleration sensor14 arranged on z axis. In this case, the roll angle theta is obtainedbased on the following mathematical formulae (3) and (4).

Here, as shown in the mathematical formula (3), different mathematicalformulae are prepared depending on positive/negative sensor displacementvalues X_(accl) and Z_(accl). As a result, it is possible to express theroll angle theta in the system of 0 to 360 degrees. However, the rangeof the function arcsin(x) is −1<x<1. So, when the acceleration sensors14 are affected by noise and the like, calculation error may occurinfrequently based on the mathematical formula (3). In this case, asshown in the following mathematical formula (4), the function arctan(x)is used. The range of the function arctan(x) is −infinity<x<infinity. Asa result, calculation error may not occur. Note that the mathematicalformula (3) and the mathematical formula (4) may be used depending onlimitation of the range, calculation accuracy of posture to becontrolled, and the like.

Next, a method of calculating the pitch angle phi will be described. Thepitch angle phi corresponds to an angle formed by the vector V and thevector V₀, and is a rotation angle around x axis of the localcoordinate. The CPU 51 calculates the pitch angle phi in the localcoordinate system with respect to the global coordinate system based onthe acceleration values and the gravity acceleration received from theinput device 10. Here, B is indicative of the displacement value withrespect to gravity acceleration 1 G output from the acceleration sensor14 arranged on y axis. Y_(accl) is indicative of the measureddisplacement value output from the acceleration sensor 14 arranged on yaxis. Z_(accl) is indicative of the displacement value output from theacceleration sensor 14 arranged on z axis. In this case, themathematical formula (5) expresses the pitch angle phi.

Further, similar to the above-mentioned mathematical formulae (3) and(4), the function arcsin(x) of the mathematical formula (5) may not beused in order to broaden the range. Instead, the function arctan(x) ofthe mathematical formula (6) may be used. As a result, calculation errormay not occur. Similar to the mathematical formulae (3) and (4), themathematical formulae (5) and (6) may be used depending on limitation ofthe range, calculation accuracy of posture to be controlled, and thelike. The CPU 51 is capable of obtaining the roll angle theta and thepitch angle phi of the input device 10 based on the mathematical formula(3) to the mathematical formula (6).

Next, the CPU 51 obtains an orientation angle and a geomagneticdepression/elevation angle based on the roll angle theta and the pitchangle phi of the input device 10, and based on magnetic intensities ofthe respective local coordinate axes (Step S3). The roll angle theta andthe pitch angle phi are obtained based on the above-mentionedcalculation. The magnetic intensities are obtained from the magneticsensors 16. As a result, inclination of the input device 10 to thedisplay apparatus 60 is determined.

First, a method of calculating the orientation angle alpha will bedescribed. X_(mag), Y_(mag), and Z_(mag) are indicative of thedisplacement values output from the magnetic sensors 16 on xyz axes ofthe local coordinate system, respectively. X_(H), Y_(H), and Z_(H) areindicative of values obtained by projecting resultant vectors ofdisplacement values, which are output from the magnetic sensors 16, tothe global coordinate XY plane. In this case, the mathematical formula(7) is derived based on the above-mentioned roll angle theta and pitchangle phi.

The CPU 51 obtains an angle (orientation) formed by the magnetic north(global coordinate system Y axis direction) and the front direction ofthe input device 10 based on the mathematical formula (7). Theorientation is transformed to an angle formed by the control target 60 band the input device 10. The transformation method will be describedlater. Here, a method of calculating the geomagneticdepression/elevation angle theta_(H) will be described. The geomagneticdepression/elevation angle theta_(H) is an angle formed by the XYhorizontal plane and the geomagnetic direction, and corresponds to anangle formed by Y axis and the vector H. The CPU 51 calculates thegeomagnetic depression/elevation angle theta_(H) of geomagnetism in theglobal coordinate system and the orientation angle of the input device10 to the display apparatus 60 based on the magnetic values in the localcoordinate system received from the input device 10. The geomagneticdepression/elevation angle theta_(H) is calculated based on themathematical formula (8), which uses the above-mentioned X_(H), Y_(H),and Z_(H).

Next, another example of calculating the geomagneticdepression/elevation angle theta_(H) will be described. The latitude phiand the longitude lambda of the position of the input device 10 arepreviously obtained. Alternatively, the latitude phi and the longitudelambda are obtained by a GPS or the like. In those cases, theta_(H) mayalso be obtained based on the following mathematical formula (9).

Note that the mathematical formula (9) is obtained by approximating thedistribution of the geomagnetic depression angle theta_(H) by thequadratic expression of latitude and longitude. The mathematical formula(9) is changed depending on movement of the magnetic pole. So themathematical formula (9) may be updated on a regular basis. As a result,the mathematical formula (9) may be accurate always. Further, a valueobtained by primary approximation of the first and second terms of themathematical formula (9) may be used, depending on accuracy required forobtaining the geomagnetic depression/elevation angle theta_(H).

Next, the CPU 51 executes primary processing for transforming theacceleration and the angular velocity detected in the local coordinatesystem into the global coordinate system, based on the orientation angleobtained from the above-mentioned calculation and based on displacementvalues output from the acceleration sensors 14 (Step S4). Thisprocessing corresponds to calculation processing in a state where theinput device 10 stands still.

First, with reference to FIG. 16 to FIG. 18, how to define axes will bedescribed. After that, the calculation processing of Step S4 will bedescribed.

FIG. 16 is an explanatory diagram showing an example of acceleration a′and angular velocity omega′ detected in the local coordinate system.

FIG. 17 is an explanatory diagram showing an example of acceleration aand angular velocity omega transformed into the global coordinatesystem.

FIG. 18 is an explanatory diagram showing an example of unit vectors i,j, k in the global coordinate system.

The CPU 51 integrates the acceleration a′ and the angular velocityomega′ detected in the local coordinate system of FIG. 16. As a result,the CPU 51 obtains velocity, travel distance, and a rotation angle ofthe input device 10. In this case, it is necessary to transform theacceleration a′ and the angular velocity omega′ in the local coordinateinto the acceleration a and the angular velocity omega in the globalcoordinate of FIG. 17.

In this case, first, as shown in FIG. 16, unit vectors of the localcoordinate system axes x, y, z are referred to as i, j, k, respectively.Further, angles formed by the unit vector i, j, k and the globalcoordinate XY plane are referred to as theta_(x), theta_(y), theta_(z),respectively (see FIG. 18).

An angle gamma of FIG. 18 is formed by the display apparatus 60, onwhich the control target 60 b is displayed, and the input device 10.Here, in FIG. 13, the display apparatus 60, on which the control target60 b is displayed, is on the global coordinate Y axis, and faces themagnetic north direction. However, from a practical point of view, thedisplay apparatus 60 is not always arranged such that the displayapparatus 60 faces the magnetic north. In view of this, the angle gammaformed by the display apparatus 60, on which the control target 60 b isdisplayed, and the input device 10 is defined by the followingmathematical formula (10) where C is indicative of the orientation, inwhich the display apparatus 60 is arranged, and where the orientationangle alpha of the input device 10 obtained based on the mathematicalformula (7) is used.

Here, the acceleration a′ and the angular velocity omega′ in the localcoordinate system of FIG. 16 may be expressed by the acceleration a andthe angular velocity omega in the global coordinate system of FIG. 17based on the following mathematical formula (11).

Here, a matrix E_(n) is a transform matrix including reference unitvectors i, j, k in the local coordinate system as elements. Further, thevectors are calculated based on theta_(x), theta_(y), theta_(z), andgamma.

First, according to the present flow, the acceleration a and the angularvelocity omega are calculated in a state where the input device 10stands still. So, theta_(x), theta_(y), theta_(z) are expressed by thefollowing mathematical formula (12). In the mathematical formula (12),alpha_(x), alpha_(y), alpha_(z) are indicative of the displacementvalues of the acceleration sensors 14 arranged on the local coordinateaxes, respectively.

In this case, the acceleration g of the acceleration sensors 14 isexpressed by the following mathematical formula (13) using alpha_(x),alpha_(y), alpha_(z).

The acceleration g of the mathematical formula (13) is a resultantvector in a case where the three-axes acceleration sensors 14 standstill. Only gravity acceleration is output. In view of this, theacceleration g is about 9.8 m/s². Further, the coordinate transformationmatrix E₀ including the reference unit vectors i, j, k as elements in acase where t=0 (when input device 10 stands still) is obtained based onthe following mathematical formula (14) using theta_(x), theta_(y),theta_(z), and gamma. theta_(x), theta_(y), theta_(z) are obtained basedon the mathematical formula (12). gamma is indicative of the angleformed by the input device 10 and the display apparatus 60, which isobtained based on the mathematical formula (10).

Next, the acceleration a′ and the angular velocity omega′ in the localcoordinate are transformed into the acceleration a and the angularvelocity omega in the global coordinate based on the coordinatetransformation matrix E₀ obtained based on the above-mentionedcalculation (Step S5).

The coordinate transformation matrix E₀ in a case where t=0 (when inputdevice 10 stands still) is obtained based on the mathematical formula(14). In view of this, the acceleration a′₀ and the angular velocityomega′₀ in the local coordinate in a case where t=0 are transformed intothe acceleration a₀ and the angular velocity omega₀ in the globalcoordinate based on the coordinate transformation matrix E₀. Thefollowing mathematical formula (15) is obtained based on themathematical formula (11).

Here, a′_(X0), a′_(Y0), a′_(Z0), omega′_(X0), omega′_(Y0), omega′_(Z0)in the mathematical formulae (15) and (16) are indicative ofdisplacement values of the acceleration sensors 14 and displacementvalues of the angular velocity sensors 15 arranged on xyz axes of thelocal coordinate system, respectively. Further, a_(X0), a_(Y0), a_(Z0),omega_(X0), omega_(Y0), omega_(Z0) are indicative of acceleration andangular velocity in XYZ axes of the global coordinate system,respectively.

Next, the CPU 51 transforms the acceleration a and the angular velocityomega in the global coordinate system, which are obtained based on theabove-mentioned calculation, into travel distance and rotation angles inthe global coordinate system, respectively (Steps S6 to S8). Thecalculation includes, for example, integral processing. Further, the CPU51 calculates velocity and travel distance of the control target 60 b inthe global coordinate system based on the acceleration of the controltarget 60 b in the global coordinate system. Further, the CPU 51calculates angles that the control target 60 b travels in the globalcoordinate system based on angular velocity of the control target 60 bin the global coordinate system. Note that, in FIG. 15, acceleration,velocity, travel distance, angular velocity, and an angle in the globalcoordinate system are referred to as global acceleration, globalvelocity, global travel distance, global angular velocity, and a globalangle, respectively.

The CPU 51 calculates the acceleration a_(X0), a_(Y0), a_(Z0) and theangular velocity omega_(X0), omega_(Y0), omega_(Z0) of the input device10 in XYZ axes of the global coordinate, respectively, based on theabove-mentioned mathematical formulae (15) and (16). Further, processingin a case where t is not 0 (described below) is executed. As a result,the CPU 51 executes coordinate transformation every time the CPU 51obtains displacement values from the various sensors, and obtains theacceleration a₀ and the angular velocity omega₀ of the input device 10in each of XYZ axes of the global coordinate. Note that, in order totransform the acceleration a₀ and angular velocity omega₀ into thetravel distance and the rotation angles of the input device 10, it isnecessary to time-integrate the acceleration a₀ and the angular velocityomega₀.

As a time-integration method, various integral processing such astrapezoidal method, midpoint method, and Simpson method are used. Thevarious integration methods will not be described in detail here.

Next, coordinate transformation (for example, t=0) is executed atarbitrarily timing after the input device 10 starts to move. After that,it is determined that if coordinate transformation of displacementvalues obtained from the various sensors (t=1) is successively executedor not at the next sampling timing (Step S9). As a result, it isunderstood that the input device 10 is moving or stops.

The acceleration a_(X0), a_(Y0), a_(Z0), and the angular velocityomega_(X0), omega_(Y0), omega_(Z0) of the input device 10 in XYZ axes ofthe global coordinate are calculated, respectively, based on themathematical formulae (15) and (16). omega′₁ is indicative of theangular velocity in the local coordinate system, which is detected bythe angular velocity sensors 15 at the next sampling timing (t=1) aftert=0. omega′_(1X), omega′_(1Y), omega′_(1Z) are indicative of therespective elements. In this case, if the following mathematical formula(17) is satisfied, then it means that the input device 10 is moving. Inthis case, the next step is calculated.

If the mathematical formula (18) is satisfied, then it means that theinput device 10 is not moving and stops. In this case, the flow returnsto the coordinate transformation calculation start determinationprocessing (Step S1). In Step S1, the mathematical formula (1) iscalculated.

Meanwhile, if the mathematical formula (17) is satisfied, the CPU 51obtains a 3D rotation matrix R^(omega theta) (Step S10). According tothe 3D rotation matrix R^(omega theta), the input device 10, which isdetermined that it is moving, rotates by the roll angle theta around thevector omega′ of FIG. 16.

The input device 10 rotates by the roll angle theta around the vectoromega′ of FIG. 16 and FIG. 17. Let's assume that DELTAt is indicative ofthe sampling cycle of displacement values from the various sensors. Inthis case, the following mathematical formula (19) is satisfied.

Next, let's assume that lambda, mu, nu are indicative of elements of theunit vectors of the vector omega′, respectively. In this case, the unitvectors are expressed by the following mathematical formula (20).

According to the mathematical formulae (19) and (20), the 3D rotationmatrix R^(omega theta) is expressed by the following mathematicalformula (21). According to the 3D rotation matrix R^(omega theta), theinput device 10 rotates by the roll angle theta around the vector omega′of FIG. 16.

Next, the CPU 51 calculates a coordinate transformation matrix (StepS11). According to the coordinate transformation matrix, accelerationand angular velocity of the input device 10 detected in the localcoordinate system are transformed into the global coordinate system.Here, the input device 10 rotates by the roll angle theta around thevector omega′. It is determined that the input device 10 is moving basedon the mathematical formula (17).

The coordinate transformation matrix E_(n) rotates by the roll angletheta around the vector omega′. In this case, the new coordinatetransformation matrix E_(n+1) is expressed by the following mathematicalformula (22) using the 3D rotation matrix R^(omega theta).

When the input device 10 is moving, the coordinate transformation matrixis updated at the sampling cycle based on the mathematical formula (22).Here, the CPU 51 updates the transformation matrix E_(n) to thetransformation matrix E_(n+1) based on the angular velocity values inthe local coordinate system received from the input device 10. Thetransformation matrix E_(n) is used to transform angular velocity of thecontrol target 60 b in the local coordinate system into angular velocityin the global coordinate system. The E_(n+1) is reflected in themathematical formulae (15) and (16). As a result, it is possible toobtain acceleration and angular velocity in the global coordinatesystem.

(how to Obtain Parallax)

Here, the control target 60 b is stereoscopically displayed in X axis ofthe global coordinate system as shown in FIG. 9 and FIG. 10. Suchstereoscopic display is realized by correlating movement of the inputdevice 10 in the global coordinate with parallax of an image displayedon the display apparatus 60.

FIG. 19 is an explanatory diagram showing positional relation of a userand the display apparatus 60 in the global coordinate system.

FIG. 20 is an explanatory diagram showing an example of user's parallaxin addition to the positional relation of FIG. 19.

L is indicative of distance between a user, who operates the inputdevice 10, and the display apparatus 60. P is indicative of imageparallax of the display apparatus 60 as shown in FIG. 20. D isindicative of image depth. E is indicative of distance between theeyeballs of a user. In this case, the following mathematical formula(23) is satisfied.

D:P=(D+L):E  (23)

This mathematical formula means the following situation. That is, thecontrol target 60 b is to be displayed as if the control target 60 bdepresses from the display apparatus 60 by 24 m (D=24 m) where E=65 mmand L=2 m, for example. In this case, P=6 cm. It means that the displayimage is to be displaced by 6 cm. Here, T is indicative of traveldistance of the input device 10 in X axis of FIG. 20. In this case, asshown in the following mathematical formula (25), the image parallax Pis adjusted by using the constant alpha. As a result, it is possible tocontrol the control target 60 b, which is a 3D image due to trick of theeye, by using the input device 10.

P=alphaT  (25)

FIG. 21 is an explanatory diagram in a case where the control target 60b is displayed as if the control target 60 b bursts from the displayapparatus 60.

Also in this case, similar to the mathematical formulae (23) and (24),calculation is executed based on the following mathematical formulae(26) and (27).

D:P=(D−L):E  (26)

The control target 60 b is to be displayed such that the control target60 b bursts from the display apparatus 60 by 1.2 m (D=1.2 m) where E=65mm and L=2 m, for example. In this case, P=10 cm. It means that thedisplay image is to be displaced by 10 cm. Further, T is indicative oftravel distance of the input device 10 in X axis of FIG. 21. In thiscase, the image parallax P is adjusted based on the constant alpha ofthe mathematical formula (25). As a result, it is possible to controlthe control target 60 b, which is a 3D image due to trick of the eye, byusing the input device 10.

As described above, the control system 100 according to this embodimentexecutes various kinds of numerical processing based on displacementvalues obtained from a plurality of sensors. In this case, a controltarget is displayed such that the control target moves to a positioncorresponding to displacement amounts in a first coordinate system basedon a transformation matrix. The transformation matrix is used totransform displacement amounts of an input device in a second coordinatesystem into displacement amounts in the first coordinate system. As aresult, a user is capable of naturally moving the control target byusing an input device in a direction that the user wishes to control thecontrol target. User-friendliness of an input device, which has nodirectionality, is improved.

Here, if the input device 10 executes the numerical processing,processing speed may be slow or calculation accuracy may be inadequate.Because of this, the input device 10 may execute only simplecalculation, i.e., Step S1. Further, the CPU 51 may execute complicatedcalculation, i.e., Steps S2 to S11. Alternatively, the input device 10only sends displacement values output from the various sensors. Further,the CPU 51 of the control apparatus 50 may execute complicatedcalculation.

Meanwhile, an operational integrated circuit having high throughput maybe mounted on the input device 10, and the input device 10 may executeall the calculation. It may be arbitrarily selected whether the inputdevice 10 executes numerical calculation processing or an apparatusdifferent from the input device 10 executes numerical calculationprocessing depending on environment such as cost, chip size, andoperational processing ability. Further, the signal transmission methodmay be realized by communication between apparatuses such as Zigbee(registered trademark) or Bluetooth (registered trademark), or bycommunication via the Internet.

Further, it is possible to conform movement of the control target 60 bto behavior of a user irrespective of directionality of the input device10, when the two-dimensionally or three-dimensionally displayed controltarget 60 b is controlled. By using the calculation method of thepresent disclosure, it is possible to minimize the gap between movementof the input device 10 and movement of the control target 60 b. It isnot necessary for a user to frequently change the direction of his gazeto check his hand, for example. As a result, the user is capable ofcontrolling the control target 60 b intuitively.

Further, the input device 10 is rotating, or a user's hand, which holdsthe input device 10, slightly shakes. In such a case, the input device10 may move by small distance. Even in such a case, the input device 10is capable of determining positional relation between the displayapparatus 60 and the input device 10, and controlling the control target60 b by itself, because the input device 10 includes the angularvelocity sensors 15.

2. Modification Examples Configuration Example of Display Apparatus

Note that, in the above-mentioned embodiment, the control apparatus 50controls movement display of the control target 60 b displayed on thedisplay apparatus 60. Alternatively, a display apparatus 70 may controlmovement display of the control target 60 b by itself. Here, aconfiguration example of the display apparatus 70 will be described.

FIGS. 22A and 22B are block diagrams showing an internal configurationexample of the display apparatus 70. FIG. 22A shows a configurationexample of the control system 100′. FIG. 22B shows an internalconfiguration example of the display apparatus 70.

The display apparatus 70 includes the display unit 60 a. The displayapparatus 70 is an all-in one computer apparatus including a display,and has functions of a general computer apparatus. A circle icon and asquare icon are displayed on the display unit 60 a as the controltargets 60 b. Further, the input device 10 in an operable status is infront of the display apparatus 70 (FIG. 22A).

The display apparatus 70 includes the display unit 60 a, a CPU 73, a ROM74, a RAM 75, a communication unit 76, and a storage unit 77.

The ROM 74 is a nonvolatile memory, and stores various programsnecessary for processing executed by the CPU 73. The RAM 75 is avolatile memory, and is used as a work area for the CPU 73.

The communication unit 76 includes an antenna (not shown) and the like,and receives various kinds of information sent from the input device 10.Further, the communication unit 76 is also capable of sending a signalto the input device 10. The CPU 73 executes processing of the respectiveunits of the display apparatus 70, and executes control similar to theabove-mentioned CPU 51 of the control apparatus 50. That is, the CPU 73controls a control target, which is displayed on the display unit 60 a,based on various kinds of information received from thesending/receiving circuit 17.

Further, the display apparatus 70 includes an operational unit (notshown). The operational unit is, for example, a keyboard. A user inputssetting such as initial setting and special setting by using theoperational unit. The operational unit receives various instructionsfrom a user, and outputs input signals to the CPU 73. As describedabove, the display apparatus 70 has an all-in-one configurationincluding the control apparatus 50 and the above-mentioned displayapparatus 60. Because of this, the configuration of the control system100′ may be simplified.

Other Modification Examples

Further, the distance between the input device 10 and the displayapparatus 60 may be arbitrarily set because the distance does not affectcontrolling performance.

For example, the input device 10 may be used as a remote control forcontrolling a robot arm, a crane, or the like. The robot arm, the crane,or the like, which moves in reality, may be displayed on the displayapparatus 60.

Further, the control target 60 b is a virtual object displayed on adisplay. It is possible to display and control the control target 60 birrespective of the kind of object. Because of this, even aninexperienced user, who is not used to a computer apparatus, is capableof controlling the control target 60 b intuitively

Further, the input device 10 has a spherical shape. Alternatively, theinput device may have a shape other than a sphere. A shape having nodirectionality such as, for example, a regular polyhedron or asemiregular polyhedron may also be employed.

Further, the input device 10 is capable of using sensor data in nineaxes. However, it is not always necessary to use sensor data of all theaxes. Partial sensor data may be extracted to be used to control thecontrol target 60 b.

Further, hardware may execute the series of processing in theabove-mentioned embodiment. Alternatively, software may execute theseries of processing. The following computer may execute the series ofprocessing. That is, programs configuring the software are installed indedicated hardware of a computer.

Alternatively, programs for executing various functions are installed ina computer. For example, programs configuring desired software may beinstalled in a general-purpose personal computer or the like, and thecomputer may execute the programs.

Further, a recording medium, in which program codes of software thatrealizes the functions of the above-mentioned embodiments are recorded,may be supplied to a system or an apparatus. Further, the system or acomputer (or controller device such as CPU) of the apparatus may readout and execute the program codes stored in the recording medium. As aresult, the functions of the above-mentioned embodiments are realized,as a matter of course.

In this case, as a recording medium in which program codes are supplied,for example, a flexible disk, a hard disk, an optical disk, amagnetooptical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatilememory card, a ROM, or the like may be used.

Further, a computer executes read-out program codes, whereby functionsof the above-mentioned embodiment are realized. In addition, an OS andthe like running in the computer execute part of or all of actualprocessing based on instructions of the program codes. This disclosurealso includes a case where functions of the above-mentioned embodimentsare realized based on the processing.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

Note that the present disclosure may adopt the following configurations.

(1) A control method, comprising:

outputting a displacement value depending on a displacement amount of aninput device in a case where operation is input in the input device, theoperation being for a control target displayed on a display apparatus,the position of the display apparatus being specified by a firstcoordinate system, the position of the input device being specified by asecond coordinate system;

calculating a displacement amount of the input device in the secondcoordinate system based on the displacement value; and

displaying the control target, the control target being moved to aposition corresponding to a displacement amount in the first coordinatesystem by using a transformation matrix, the transformation matrixtransforming a displacement amount of the input device in the secondcoordinate system into a displacement amount in the first coordinatesystem.

(2) The control method according to (1), further comprising

updating a transformation matrix based on an angular velocity value inthe second coordinate system, the transformation matrix transformingangular velocity in the second coordinate system into angular velocityof the control target in the first coordinate system, the angularvelocity value in the second coordinate system being received from aninput device, the input device including an input device main body andan angular velocity sensor, the angular velocity sensor beingaccommodated in the input device main body, the angular velocity sensorbeing configured to detect angular velocity of the input device mainbody and to output the displacement value, the displacement valueincluding an angular velocity value depending on the detected angularvelocity.

(3) The control method according to (1) or (2), further comprising:

calculating a roll angle based on an acceleration value in the secondcoordinate system, the roll angle being an angle of rotation of theinput device in a direction that the input device faces the displayapparatus, the acceleration value in the second coordinate system beingreceived from the input device, the input device including anacceleration sensor, the acceleration sensor being accommodated in theinput device main body, the acceleration sensor being configured todetect acceleration and gravity acceleration of the input device mainbody and to output the displacement value, the displacement valueincluding an acceleration value, the acceleration value depending on thedetected acceleration and the detected gravity acceleration; and

calculating a pitch angle of the second coordinate system with respectto the first coordinate system based on the acceleration value and thegravity acceleration.

(4) The control method according to any one of (1) to (3), wherein

the transformation matrix transforms the acceleration value and theangular velocity value in the second coordinate system into accelerationand angular velocity of the control target in the first coordinatesystem.

(5) The control method according to any one of (1) to (4), furthercomprising

calculating a geomagnetic depression/elevation angle in the firstcoordinate system and an orientation angle of the input device withrespect to the display apparatus based on a magnetic value in the secondcoordinate system, the magnetic value in the second coordinate systembeing received from the input device, the input device including amagnetic sensor, the magnetic sensor being accommodated in the inputdevice main body, the magnetic sensor being configured to detectgeomagnetic orientation and to output the displacement value, thedisplacement value including a magnetic value depending on the detectedgeomagnetic orientation.

(6) The control method according to any one of (1) to (5), furthercomprising:

calculating velocity and travel distance of the control target in thefirst coordinate system based on acceleration of the control target inthe first coordinate system; and

calculating a travel angle of the control target in the first coordinatesystem based on angular velocity of the control target in the firstcoordinate system.

(7) A control apparatus, comprising:

a calculation unit configured to calculate a displacement amount of aninput device in a second coordinate system based on a displacementvalue, the displacement value being output depending on the displacementamount of the input device in a case where operation is input in theinput device, the operation being for a control target displayed on adisplay apparatus, the position of the display apparatus being specifiedby a first coordinate system, the position of the input device beingspecified by the second coordinate system; and

a display controller unit configured to display the control target, thecontrol target being moved to a position corresponding to a displacementamount in the first coordinate system by using a transformation matrix,the transformation matrix transforming a displacement amount of theinput device in the second coordinate system into a displacement amountin the first coordinate system.

(8) A program, causing a computer to execute the steps of:

outputting a displacement value depending on a displacement amount of aninput device in a case where operation is input in the input device, theoperation being for a control target displayed on a display apparatus,the position of the display apparatus being specified by a firstcoordinate system, the position of the input device being specified by asecond coordinate system;

calculating a displacement amount of the input device in the secondcoordinate system based on the displacement value; and

displaying the control target, the control target being moved to aposition corresponding to a displacement amount in the first coordinatesystem by using a transformation matrix, the transformation matrixtransforming a displacement amount of the input device in the secondcoordinate system into a displacement amount in the first coordinatesystem.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2012-081549 filedin the Japan Patent Office on Mar. 30, 2012, the entire content of whichis hereby incorporated by reference. It should be understood by thoseskilled in the art that various modifications, combinations,sub-combinations and alterations may occur depending on designrequirements and other factors insofar as they are within the scope ofthe appended claims or the equivalents thereof.

REFERENCE SIGNS LIST

-   10 input device-   11 CPU-   14 acceleration sensor-   15 angular velocity sensor-   16 magnetic sensor-   20 input device main body-   50 control apparatus-   60 display apparatus-   100 control system

1. A control method, comprising: outputting a displacement valuedepending on a displacement amount of an input device in a case whereoperation is input in the input device, the operation being for acontrol target displayed on a display apparatus, the position of thedisplay apparatus being specified by a first coordinate system, theposition of the input device being specified by a second coordinatesystem; calculating a displacement amount of the input device in thesecond coordinate system based on the displacement value; and displayingthe control target, the control target being moved to a positioncorresponding to a displacement amount in the first coordinate system byusing a transformation matrix, the transformation matrix transforming adisplacement amount of the input device in the second coordinate systeminto a displacement amount in the first coordinate system.
 2. Thecontrol method according to claim 1, further comprising updating atransformation matrix based on an angular velocity value in the secondcoordinate system, the transformation matrix transforming angularvelocity in the second coordinate system into angular velocity of thecontrol target in the first coordinate system, the angular velocityvalue in the second coordinate system being received from an inputdevice, the input device including an input device main body and anangular velocity sensor, the angular velocity sensor being accommodatedin the input device main body, the angular velocity sensor beingconfigured to detect angular velocity of the input device main body andto output the displacement value, the displacement value including anangular velocity value depending on the detected angular velocity. 3.The control method according to claim 2, further comprising: calculatinga roll angle based on an acceleration value in the second coordinatesystem, the roll angle being an angle of rotation of the input device ina direction that the input device faces the display apparatus, theacceleration value in the second coordinate system being received fromthe input device, the input device including an acceleration sensor, theacceleration sensor being accommodated in the input device main body,the acceleration sensor being configured to detect acceleration andgravity acceleration of the input device main body and to output thedisplacement value, the displacement value including an accelerationvalue, the acceleration value depending on the detected acceleration andthe detected gravity acceleration; and calculating a pitch angle of thesecond coordinate system with respect to the first coordinate systembased on the acceleration value and the gravity acceleration.
 4. Thecontrol method according to claim 3, wherein the transformation matrixtransforms the acceleration value and the angular velocity value in thesecond coordinate system into acceleration and angular velocity of thecontrol target in the first coordinate system.
 5. The control methodaccording to claim 4, further comprising calculating a geomagneticdepression/elevation angle in the first coordinate system and anorientation angle of the input device with respect to the displayapparatus based on a magnetic value in the second coordinate system, themagnetic value in the second coordinate system being received from theinput device, the input device including a magnetic sensor, the magneticsensor being accommodated in the input device main body, the magneticsensor being configured to detect geomagnetic orientation and to outputthe displacement value, the displacement value including a magneticvalue depending on the detected geomagnetic orientation.
 6. The controlmethod according to claim 5, further comprising: calculating velocityand travel distance of the control target in the first coordinate systembased on acceleration of the control target in the first coordinatesystem; and calculating a travel angle of the control target in thefirst coordinate system based on angular velocity of the control targetin the first coordinate system.
 7. A control apparatus, comprising: acalculation unit configured to calculate a displacement amount of aninput device in a second coordinate system based on a displacementvalue, the displacement value being output depending on the displacementamount of the input device in a case where operation is input in theinput device, the operation being for a control target displayed on adisplay apparatus, the position of the display apparatus being specifiedby a first coordinate system, the position of the input device beingspecified by the second coordinate system; and a display controller unitconfigured to display the control target, the control target being movedto a position corresponding to a displacement amount in the firstcoordinate system by using a transformation matrix, the transformationmatrix transforming a displacement amount of the input device in thesecond coordinate system into a displacement amount in the firstcoordinate system.
 8. A program, causing a computer to execute the stepsof: outputting a displacement value depending on a displacement amountof an input device in a case where operation is input in the inputdevice, the operation being for a control target displayed on a displayapparatus, the position of the display apparatus being specified by afirst coordinate system, the position of the input device beingspecified by a second coordinate system; calculating a displacementamount of the input device in the second coordinate system based on thedisplacement value; and displaying the control target, the controltarget being moved to a position corresponding to a displacement amountin the first coordinate system by using a transformation matrix, thetransformation matrix transforming a displacement amount of the inputdevice in the second coordinate system into a displacement amount in thefirst coordinate system.